APR. 18. 2006 9:22AM HP LEGAL 



NO. 330 P. 11 



(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europtcfl des brevets 



(12) 



(43) Date of publication; 

26.04.1999 Bulletin 7 

(21) Application number 90308506^ 

(22) Data of filing: 16.10-1998 



en) EP0 911 759 A2 

EUROPEAN PATENT APPLICATION 

(51) intCL 6 : GO 6T 7/40 



(84) D9signated Contracting States: 


* Kon slant In idea, {Constantino? 


AT BE CH CY DE DK ES R FR QB GR IE IT LI LU 


San Jose, California 951 1 8 (US) 


MC NL PT SE 


» Trotter, Daniel R. 


Designated Extension Stales: 


Mountain View, California 94040 (US) 


ALLTLVMK ROSI 


• Parry, Andrew 




Sunnyvale, California 94304 (US) 


(30) Priority: 23.10.1997 US 956380 




(74) Representative: Jehan, Robert ctal 


(71) Applicant Hewlett-Packard Company 


William*, Powell & Associates, 


Palo Alto, California 34304 (US) 


4 St Paul's Churchyard 


London EC4M GAY (GB) 


(72) Inventors: 




• Lin,qlan 




Santa Clara, California 95051 (US) 





(54) apparatus and a method for reducing red-eye in an image 



(57) A red-eye reduction system (10) includes a 
masking modufe (16) which converts an image (1 1) into 
a mask having first state areas representing red colour 
pixels of the Image and second state areas representing 
other colour pixels of the image. The image (H ) includes 
an eye with a red pupil. A pupil locating module (17) is 
coupled to the masking module (16) to locate a substan- 
tially first state area in the mask that resembles a pupil. 



A colour replacing module (1S) is then coupled to the 
pupil locatng module (17) to change the red colour pix- 
els in the area into monochrome (grey) or other prede- 
fined colours. The colour replacing module (18) also ad- 
justs the boundary of the area by changing the colours 
of pixels in close proximity to the area if the colour of 
these pixels is determined to be sufficiently close to red 
such that natural appearance of the eye is maintained 
when reducing the rod pupil. 
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Description 

[0001] The present invention pertains to digital image processing. More particularly, this invention relates to auto- 
matically reducing red-eye in a digital image while maintaining the natural appearance of the eyes. 

5 [0002] As is known, photographing a person In a relatfvery dark environment requires flashlight to avoid under-ex- 
posure. The use ot flashlight, however, often results in a person's eyes being red in the photograph, making the p&rsco 
look ■spooky" or unnatural in the photograph. This is typically referred to as the •red-eye" phenomenon or simply red- 
eye. , 
[0003] The red-eye typically resu Its from the person's pupils not being able to quickly adjust to the flashlight in dark- 

io ness. As is known, the pupils of a person are enlarged in a dark environment. When flashlight appears, the pupils are 
not able to reduce their si2es due to the suddenness of the flashlight. This typically causes the flashlight reflecting off 
the retina at the back of the eyes, causing red-eye. 

[0004] Several prior art techniques have been proposed to reduce the red-eye effect. A common prior art approach 
is to uso multiple flashes in the camera to ccnTract the pupils before a final flash is used to expose and capture the 

is image. However disadvantages are associated with this prior art approach. One disadvantage is the delay between 
the time when the first flashlight appears and the time when the picture is actually taken. This means the picture is 
taken several seconds after the exposure button has been pressed. This may cause confusion and the subjects may 
move away from the posed positions before the image is captured. Moreover, the red-eye problem still occurs when 
the user forgets to enable this feature of the camera during photographing, or when the camera is not equipped with 

so such red-eye prevention feature. Further, this prior art approach cannot soke the red-eye problem in the already-taken 
photos. 

[0005] With the advance ot image processing technologies, it is now possible to digitize an image and store the 
digitized image in a compute r system. This is typically done either using a digital camera to capture the image digitally 
or using a scanner that converts the image into digital form. The digital image Includes data representing image pixels 
25 arranged in a matrix. The data ot the digital image are then stored In the computer. The digital image can be retrieved 
for display and can also be digitally altered in the computer. 

[00061 Because images can now be captured as or converted into digital images, it is thus possible to correct the 
red-eye problem in an image digitally. Some prior art schemes have been proposed to correct the red-eye problem 
digitally. One such prior art scheme simply provides the user with means for manually painting over the red eyes 
3d digitally. Trie disadvantage of this prior art scheme is that some kind of painting skill is needed for the user to paint 
over the red eyes. Another disadvantage is that the correction of the red-eye is not done automatically, but rather 
manually. 

[0007] Another prior art approach requires the user to precisely locate the center of a pupil so that a black circle is 
placed over the red-eye region. The disadvantage of this prior art approach is that the red-eye region b often not a 

ss circular region. This may cause portions of the red-eye region not to be covered by the black circle. In addition, the 
black circle may not be able to cover the peripheral area ft©., the pink ring) of the red-eye region. Moreover, replacing 
the red pupil with a compete black circle may also cover the glint in the pupil. As Is known, the glint in the pupil is 
usually a bright "white" spot. Thus, the result of this type of correction is often quite noticeable and undesirable, and 
sometimes destroys the natural appearance of the eyes in the Image. 

40 [OOOB] The present invention seeks to provide an Improved red-eye reduction system. 

[0006] According to an aspect of the present invention, there is provided a red-eye reduction system as specified in 
claim 1 . 

[001 0] According to another aspect of the present invention, there is provided a method of reducing red-eye effect 
as specified in claim 9. 

4S [0011] Features of the preferred embodiment include to reduce red-eye in a digital image, automatically to reduce 
red-eye in an image with minimal user intervention, to reduce red-eye in an image while preserving the natural appear- 
ance of the eyes and/or to provide a robust and user-friendly arrangement for correcting red-eye. 
[001 2J A red-eye reduction system is described which includes a masking module. The masking module converts 
an image into a mask having first stale regions representing red color pixels of the image and second state regions 

£0 representing other color pixels of the image. The image includes an eye with a red pupil A pupil krating module is 
coupled to the masking module to locate a substantially first state area in Ihe mask that resembles a pupil. A color 
replacing module is coupled to the pupil locating module to change red color pixels in the area into monochrome (grey) 
or other predefined colors. The color replacing module also adjusts the boundary of the area by changing the colors 
of pixels in close proximity to the area if the color of these pixels is determined to be sufficiently close to red. 

55 [001 3] A method of reducing red-eye in a digital image is described. The method includes the step of converting an 
image having an eye with a red pupil bto a mask having first state areas representing red color pixels of the image 
and second state areas representing other color pixels of the image. The mask is then processed to locate a substan- 
tially first stale area in the mask that resembles a pupil. The method then changes the red color pixels in the area into 
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monochrome (grey) or other predefined colors With boundary adjustments when the color of pixels in close proximity 
to the area is determined to be sufficiently close to red. 

[0014] An embodiment or the present invention is described below, by way of example only, with reference to the 
accompanying drawings, in which: 
s [001 5] Figure 1 shows a red-eye reduction system that implements one embodiment of the present invention, wherein 
the red-eye reduction system includes a userimerf ace, a masking module, a pupiMocating module, and a color replacing 
module. 

[0016] Figure 2 shows the structure of the user interface. 

[0017] Figure 3 shows an image displayed by the user interlace with user defined areas that contain red eyes. 
to [001 8] Figure 4 shows the process of the masking module in converting one defined area into a binary mask that 
indicates possible location of the red pupil. 

[001 9] Figure 5 is the flow chart illustrating the operation of th s masking module. 
[0020] Figures 6 and 7 show how the pupil locating module locates the pupil area within a binary mask. 
[0021] Figure B shows the binary mask of Figure 4 after being processed by the pupil locating module. 
is [0022] Figure 9 is the flow chart illustrating the operation of the pupil locating module. 
[0023] Figure 10 is the flow chart showing the operation of the color replacing module. 

[0024] Figure 1 shows an embodiment of red-eye reduction system 10 which reduces or minimizes red-eye in an 
image once the user has marked the area within which a red pupil is located. The red-eye reduction system 1 0 then 
automatically reduces the red-eye while maintaining the natural appearance of the eye. 

20 [0025] As win be described in more detail below, the red-eye reduction system 10 includes a user interface 15 that 
allows the user of the red-eye reduction system 1 0 to mark or define an area to be processed by the red-eye reduction 
system 10 to reduce the red-eye effect. The area contains a red pupil in addition, the red-eye reduction system 10 
includes a masking module 16 that converts the marked area containing the red pupil into a binary mask of first state 
(e.g.» white) areas and second state (e.g., black) areas. The ■white 11 areas of the binary mask represent the red color 

2$ pixels within the marked area of the image. Th© "black' areas represent other color pbceis within the marked area of 
the image. 

[0026] The red-eye reduction system 1 0 also includes a pupil locating module 1 7 thai locates a substantially ^hite" 
area in the binary mask that resembles a pupiL Trie red-eye reduction system 10 further includes a color replacing 
module 1 6 that only changes the red color pbceis in the area into monochrome (grey) or other predefined colors. This 

so maintains the glint in the pupil which is typically a bright non-red spot In addition, the color replacing module IB also 
adjusts the boundary of the area by changing colors of pixels in dose proximity of the area to monochrome (grey) or 
other predefined colors if the color of those pixels is detennined to be sufficiently close red. The red-eye reduction 
system 10 will be described in more detail below, afeo in conjunction with Figures 1 through 10. 
[0027] Trie red-eye reduction system 1 0 can be implemented by software, hardware, or firmware- This means each 

3S of the modules 15-18 can be implemented by software, hardware, orfirmware. In one embodiment, the red-eye reduc- 
tion system 1 0 is a software application program that is run on a computer system (not shown). The computer system 
that runs the red-eye reduction system 10 can be a personal computer, a mini computer, a workstation computer, a 
notebook computer, a network computer, server computer, or any other data processing system Alternatively, one or 
more modules of the red-eye reduction system 10 can be implemented by hardware or firmware while others can still 

AO be software modules, 

[002S] Moreover the red-eye reduction system 10 can be implemented In an imaging system (not shown). The 
imaging system can be an image display system, a scan-to-print system, or simply a computer system having a display. 
The imaging system can also be other type of imaging system 

[0029] As is shown irr Figure 1 , the user interface 1 5 of the red-eye reduction system 1 0 is used to display an original 
45 image 11 . The original image 11 has red eyes or red pupils. Figure 3 shows one such image, as an example. As can 
be seen from Figure 3. each of the eyes 35-36 has a pupil (i.e.. pupil 38 or 39) that is red. Figure 3 also shows two 
marked or defined areas 31 and 32. in one embodiment, the marked areas 31-32 are marked by the user through the 
user interface 15. The marked areas 31 -32 will be described in more detail below 

[0030] As can be seen from Figure 3. each of the marked areas 31-32 is a rectangular region that surrounds a red 
so pupil (e.g., the pupil 33 or 39) which is to be recolored to a neutral dark). It is, however, to be noted that the marked 
area (31 or 32) need not be rectangular, but may be, in general, a polygon or have a curvilinear shape. The rectangular 
shape is shown merely asan example. The image within the marked area (e-g.. the marked area 31 or 32) is represented 
as A={a^ % wherein a$ represents a pixel within the Image A 

[0031] Referring back to Figure 1, the original Image 11 is a digital image which can be obtained from an image 
ss capturing device. Known technologies are ueedto obtain the original image 11. For example, the original image 11 can 
be produced by a scanner or a digital camera, and is directed to be displayed by a display (e.g., CRT, LCD, or flat 
panel display). Alternatively, the original image 11 may be directed to be printed by a printer. 
[0032] The user interface 15 of Figure 1 is displayed on a display (not shown). The display that displays the user 
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Interface 15 can be a computer display; The user interface IS is an interactive interface that allows the user to mark 
or define art area within the displayed original image 11 (e.g., the marked areas 31 and 32 in Figure 3) that contains 
a red eye or red pupil, the user interface 1 5 also allows the user to activate the red-eye reduction system 1 0 by placing 
a cursor on an icon displayed on the user interface 15 using a mouse or other cursor control device and then clicking 
s on the signal button of the mouse or the cursor control device to activate the red-eye reduction system 10. Figure 2 
shows the structure of the pser interface 15 which will be described in more detail below: 

[0030] As can be seen from Figure 2, the user interface 1 5 includes a tool bar or menu bar area 21 that includes an 
icon 23 for activating the red-eye reduction system 1 0 of Figure 1 , The user interface 1 5 also includes a display area 
22 within which the original image 11 is displayed. The user interface 15also includes a cursor 24 that allows the user 
io to interact with the user interface 1 5 through a mouse or other cursor control device (not shown). The cursor 24 also 
allows the user to define the marked area (e.g., the marked area 31 or 32) of the image 11 displayed. Once the red- 
eye reduction system 10 is activated through the icon 23, the red-eye reduction system 10 reduces or minimizes the 
red eye contained in the marked area. The user interface 15 can be implemented using any known user interactive 
interface technology. 

is [0034] Referring back to Figure 1 , the color image A of the marked area (e.g., The marked area 31 or 32) of the 
original image 11 is then applied to the masking module 16 of the red-eye reduction system 10. The masking module 
1 6 is used to read the color image A of the marked area and then convert the color image pixel data into a binarv mask, 
which ts represented as B={ttf, wherein 6- represents the mask value f? # for a pixel (i. j) within the binarv mask B. 
Because the mask 8 is a binary mask, each of the pixels is thus either a "white* pixel or a *black" pixel. This 

20 indicates that th e binary mask & can be of the same size'as the marked area A of the original image 1 1 . Alternatively, 
the size of the binary mask B is smaller than the marked area A The binary mask B is used to identify all the red pixels 
in the marked area A and therefore is used tor the pupil locating module 17 to identify the size and location of the red 
pupil within the marked area A 

[0035] The binary mask B includes first state areas or regions that indicate red color pixels in the marked area of the 
25 image 11 and second state areas or regions thai indicate pixels having colors other than red. The first state areas can 
be represented by. for example, ■white" (I.e., thehighest grey scale value) pixels and the second stateareas represented 
by 'black 8 (i.e.. the iowsst grey scale value) pixels. When the grey scale value is represented by an eight-bit data, the 
highest grey scale value is 255 and the lowest grey scale value is 2ero. Figure 4 shows the converted binary mask 44 
of a marked area similar to the area 31 of the original image 11 from the masking module 16, as well as the process 
so at creating the binary mask 44 by the masking module 1 6. which will be described in more detail below 

[0036J As can be seen from Figure 4, the color image A \n the marked area 31 of Figure 3 is formed by a red image 
41a, a green image 41b, and a blue image 41 c. The conversion process of the masking module 16 generally includes 
three steps. First, the masking module 16 converts the images 4la-4lc into a luminance-chrominance representation 
(i.e.. images 42a-42c). Tnis is done using known techniques. The chrominance information 42b-42c is then used to 
$5 calculate a threshold value Toetween the first state and the second state. Any known threshold determining technique 
can be used to calculate the threshold T. As is known, a large variety of threshold determining techniques have been 
devised tor image segmentation to distinguish the image foreground from the background or to identify objects in the 
image. 

[0037] Tne threshold value 7is then applied to each image pixel of the color image A in the marked area 31 to create 
ao the binary mask 44. The masking module 16 can then reduce the size of the binary mask 44 to eliminate areas that 
are unnecessary and extra for the red-eye reduction within the binary mask 44. This makes the binary mask 44 to only 
include the red pupil and a predetermined peripheral ol the red pupil. Alternatively, this step can be performed before 
the masking module 1 6 converts the image of the marked area 31 into the binary mask 44. 
[0038] In one embodiment, a nonstandard luminance^hrominance representation is employed for the iuminance- 
4& chrominance representation. In this case, the chrominance infonnation includes a red-green chrominance Cr and a 
blue-yellow chrominance Cb. Alternatively, other types of luminance-chrominance representation can be used. For 
example, the luminance-chrominance representation can be a standard luminance-chrominance representation. 
[0039] As can be seen from Figure 4, the image 42a shows the luminance representation, the image 42b shows the 
chrominance Cr representation and the image 42c shows the chrominance Cb representation. The masking module 
so 1 6 uses k a non-linear power function followed by an RGB-to-YCrCb conversion matrix to convert the images 41 a-41 c 
into the images 42a-42c The transformation Is implemented using known technology. 

[0040] The RGB represents red. blue, and green color information. Y represents the luminance information and Cr 
and Cb represent the chrominance information. Before the matrixing operation, the masking module 16 raises each 
of the RGB values of the image data A of the marked area 31 to the 1/3 power. Since the original RGB data is assumed 
«p • to have already been gamma-corrected, this gives an effective gamma of about 6.6 to the image data. The purpose 
of the 1/3 power mapping is to increase the separation between red and non-red pixels. 
[0041] In one embodiment, the threshold value 7"is computed by the following equation: 
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T *^ av9 +0 -2x(Cr ftl ^-Cr min ) 

wherein Cr avg repressnts the average chrominance Cr value. Cr^and Cr mfn represent the maximum and minimum 
s chrominance Or values, respectively. The average value, the maximum value, and the minimum value are all computed 
from the image data A of the marked area 31 . Here, the variables Cr and Co refer to the two chrominance components 
in the nonstandard luminance^chrominance space. The chrominance component Crjj for a pixel (i, j) in the marked 
area 31 is then compared with the threshold value 7: If the sum is greater than the threshold value 7; the mask value 
bp for that pixel (i, j) is one (i.e.. white). This means that the color of the pixel (L j) red Otherwise, the mask value 
10 of the pixel (i,j) is zero (i.e.. black). This results in the binary mask 44. as shown in Figure 4. 

[0042] Once the binary mask 44 is constructed, the masking module 1 6 can use a neighborhood calculation to elim- 
inate isolated red pixels and fill in isolated norwed pixels in the mask 44. Alternatively, the masking module 16 does 
not perform this step and the pupil locating module 17 performs this step. Tne neighborhood calculation can be ex- 
pressed as 

is 

bi j =l-if , I , Ib kra >2 

k-i-1 m-j-l 

20 

else, bj j = 0 

[0043) Figure S shows the process of converting the image A of the marked area of the originaJ image 11 into the 
binary mask B by the masking module 1 6 of Figure t . As can be seen from Figure 5, the process starts at the step 50. 
At the step 51 , the masking module 16 receives the color image data A from the user interlace 15, At the step 52, the 

2* masking module 16 uses the RGB-to-YCrCb conversion to convert the color image of the marked area A into tne 
luminance-chrominance representation (e.g. > the images 42a-42c), Then the masking module 16 usesthe chrominance 
information from the luminance-chrominance representation to compute the threshold value at the Step 53. At the step 
U, the masking module 1 6 constructs the binary mask B (e.g., the binary mask 44) by comparing each image pixel of 
the marked area A with the threshold value. Trie process then ends at the step 55. 

50 [0044] Referring back to Figure 1 , the binary maskBfrom the masking module 16 is then applied to the pupil locating 
module 17. The pupil locating module 17 processes the binary mask Sto locate the red pupil in the image A The pupil 
locating module 1 7 does this by locating a largest "white" area in the binary mask a This means we assume that the 
largest white area in the binary mask B is the red pupil The pupfl locating module 1 7 then generates a modified binary 
mask P= {p$ from the binary mask B. The modified binary mask P indicates the location of the red pupil in the mask, 

& The modified binary mask P is sent to the color replacing module 18. The modified b'marv mask P can be a "clean ed 
up" version of the binary mask B. This means that all other white areas in the binary mask B are converted to black 
areas as they have nothing to do with the red pupil. Alternatively, the pupil beating module 17 only determines the size 
and location of the red pupil in the binary mask 3 and does not modify the binary mask B. In this case, the modified 
binary mask P is only the binary mask B with location and size information of the red pupil in the mask 

40 [0O4S] in one embodiment, the pupil locating module 17 employs an exhaustive search and block-matching estima- 
tion scheme to locate the largest area in the binary mask B that best matches a white circular area. This exhaustive 
search scheme searches tne entire binary mask B for the largest substantially white area. The exhaustive search 
scheme will be described in more detail below, also in conjunction with Figures 6-7. Alternatively, other schemes may 
be employed by the pupil locating module 1 7 to locate the red pupil in the binary mask B. For example, the pupil locating 

45 module 17 may employ a tree-search or logarithmic search scheme to tocate the red pupiL Moreover.the pupil locating 
module 17 may apply a Hough transform scheme to locale the red pupil in the binary mask B. Using this scheme, the 
pupil locating module 17 first detects the outlines or boundaries of the white regions in the binary mask B. Then the 
pupil locating module 17 uses ths Hough transform to detect the presence of a circle among the white regions. The 
above described schemes are all known schemes and can readily be realized by software programs. 

so [0046] The operation of the pupil locating module 17 is now described in more detail below* also in conjunction with 
Figures 6-9. Suppose that the binary mask 3 received in the pupil locating module 17 is a wx h binary mask 60 (shown 
in Figure 6). In this ma3k 60. the pixel at the upper left comer is marked as the pixel (0, 0) and the pixel at the lower 
right comer is marked as the pixel (uH n-J). The mask 60 contains several white areas (i.e., white areas 61a through 
61 g). To locate the largest white area (i.e., the area 61 a) in the binary mask 60 that resembles a pupil, the pupil locating 

55 module 1 7 uses a block-matching estimation scheme to determine whether a largest white area in the binary mask 60 
better matches the characteristics of a pupil model under a given criterion. As is known, a pupil typically has a circular 
shape. Thus, a possible pupil model pm can be represented as {r, x. j/J, wherein r is the radius of the circle and (x, >) 
denote the position of the center of the circle. To simplify the computation, a square pupfl model can be used. The 
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square pupil model has paramete rs is, {s, x, y\ t wherein e denotes the width of the square and (x >) denote the position 
of ths left upper comer. The block-matching estimation scheme is a known scheme, and therefore will not be described 
in more detail. 

[0047] Using the exhaustive search scheme, the pupil locating module 17 locates the largest white area In the mask 
£ 60 that resembles the pupil model in accordance with the following routine. 

[0048] Let e =min(w, h), M = -(s?) t x and y denote the upper loft comer of a square white area. Then perform the 
following: 

.10 do { 

M old -M 

for v = 0 toy = h-s 

15 

JC = 0 to JT = w-s 

Find the s *s white area with the maximum number of 
white pixels n. 

25 ifM>M ol4 

then make the white area as the best pupii model. 

S - S - I 

SO 



) while M > Moy. 

35 

Thfe can be shown to yield the best match in the maximum likelihood sense. 

[0049] Once the pupil locating module 1 7 determines the square parameters {s, x, >} of the largest white area, the 
circular parameters {r, x. y} of that area can then be determined This can be seen tram Figure 7, which shows the 
square 62 located by the pupil locating module 17 within the white area 61 a of the mask €0. As can be seen from • 
40 Figure 7, when the parameters {s,, jc, , y, ,} of the square 62 are dete rmined, the circular parameters of the white area 
61a can be given as {n Xq, y 0l j. wherein /represents the radius and >fe and y 0 denote the center of the circle. In this 
case, ris equal to s 1 / xq is equal to + s,/2, andy 0 is equal to y y + s n /2. Alternatively, rcan be made to equal to 
s, 12. Figure 8 shows a modified binary mask 80 generated by the pupil locating module 1 7 from the binary mask 44 
of Figure 4. 

& [0050] Figure 9 shows the main steps of the pupil locating module 1 7 to locate the location and size of the red pupil 
in the binary mask ft As can be seen from Figure 9 t the process starts at the step 90. At the step 91 , the pupil locating 
module 1 7 receives the binary mask Strom the masking module 16. At the step 92, the pupil locating module 17 locates 
the location of the red pupil in the mask by locating the largest "white* area in the mask B. At the step 93, the pupil 
locating module 17 outputs the modified binarv mask P where only the white pixels of the area found in the step 92 

£0 remain white. The remaining pixels in the binary mask Pare changed to black. The process ends at the step 94. 

{0051] Referring back to Figure 1 , the color replacing module 16 is used to replace the red color pixels within the 
white area that resembles a pupil in the modified binary mask P into monochrome (grey) or other predefined colors. 
The binary mask P has a substantially white area which is determined by the pupil locating module 1 7 as the red pupa 
This area may include a black island (see Figure 9) that indicates the glint of the eye. The color replacing module 18 

55 only replaces colors of the pbcele for which the mask value pj=1. In this way, the glint of the pupil is maintained (the 
glint is a bright spot in the pupil}. The gllnl is shown in the binary mask 5 or P as a 'black* island surrounded by white 
pbcels. In one embodiment, the color data is changed so the chrominance values of ail the "white" pixels become zero. 
In addition, the cofor replacing module 1S also changes the luminance value of all pixels within the white area by a 
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predetermined factor value. In one embodiment, this factor value is approximately 0.8. 

[0052] In addition, because a red pupil typically has a pink colored region or ring outside the red pupil, rt is desirable 
to change the color of those pink pixels to maintain a natural appearance of the eye after the color of these red pupil 
pixels has been changed. However, because the color of these pink color pixels is not red enough, they are not con- 
5 verted Into the "white" pixels by the masking module 16 in the binary mask B. Thus, the color replacing module 18 
needs to "grow" the white area in the modified binary mask P in order to cover these pink color pixels. This is also 
referred to as boundary adjustment. 

[0053] To accomplish the boundary adjustment, the color replacing module 78 analyzes the color of all adjacent 
pixels in the image A that are outside the white area in the modified binary mask P, but each of which has at least one 

jfr neighboring pixel within the white area. The color replacing module 1 B analyzesthe color of adjacent pixels to determine 
if their color is sufficiently close to red. This closeness indicates a deviation in perceived redness. The color replacing 
module 18 does this by comparing the color value of an adjacent pixel with the median red value r^of all the pixels 
within the white area of the binary mask P. The color of the adjacent pixel is also compared with the difference of the 
median red value t m9tS and the median green value of all the pixels within the white area By using the median 

is values fmtf and g mofJr the glint information within the white area is effectively avoided so long as the glint pixels do not 
dominate the white area. AKematfveiy. average red and green values can be used instead of median values. 
[0054] In one embodiment and as a measure of perceived change in redness, two quantities are used for the com- 
parison. One is the percent deviation of the red value of the adjacent pixel e y with respect to the median red value r mo ^ 
which is denoted Thus, rf- = | - r^J/r^ x 1 00% . indicates the red value of the adjacent pixel a, 7 The other 

so quantity is the percent deviation in red-green difference, jMpfch is denoted as rf^jr Thus, if we define A 9 = - g^ 
and A* 5 ™ = [r^ - g m J , then t^- Is equal to - - a^/a'^ x 100%. When both t^- and rf*g are below certain 
predetermined threshold values, the color of the adjacent pixel is determined to be sufficient close to red and the 
pixel be included in the white area. 

[0055] In one embodiment, the threshold value TH r f or -n/^ is approximately thirty and the threshold value TH^ for 
25 is approximately forty. Alternatively, different threshold values can be employed for the two measurements. 

[0056] As all the adjacent pixels to the white area are analyzed the results are stored in a second mask F, A value 
P '~= \ indicates that the pixel s„ should be included in the white area of P, while a zero value means it should not After 
the mask P is constnicted, the two binary mask areas Pand P are merged to form a new version of P. This process 
can be repeated N times, wherein N is a predetermined number (e.g„ 5). The value of N dictates how far the white 
00 area of Pcan grow in attempting to make it conform to the actual shape of red pixels within the red pupil. The region 
growing process of the color replacing module 18 is expressed in pseudo-code as: initialize all 0 



35 



40 



45 



SO 



for (* = 0; 

for - (all pfxels ^-bordering {/? f> -} = 1){ 
compute x\ r £ 
Compute tp^; 

If ((if, < TH r ) AND (rT* ff < TH^ 

} 

for (all ij) 
} wherein the OR is a binary operator. 

[0057] Figure 10 shows the process of the color replacing module 18. As can be seen from Figure 10. the process 
starts at the step 100. The step 101 is an initialization step at which the variable n is set to one and me mask P 1 Is set 
to zero. At the step 102. n is compared with the predetermined number N .If n is greater than N, then the step 111 is 
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performed Otherwise, the step 103 is performed at which n is incremented. Atthestep 104. the color replacing module 
18 receives one pixel p.from the mask P. Then the step 1 05 is performed to determine if the pixel is a black pixel (i. 
e a- oi If not the step 109 is performed. If the answer is positive, then the color replacing module 1 8 checks rf any 
n«-anboring pixel of the pbcel is a We' pbcel in the mask P at the step 106. If not, the step 109 is the next step. 

s Otherwise the step 107 is performed to determine if the color of the pixel a ff is sufficiently red. If not. the step 109 is 
the next step. Otherwise, the color replacing module t8 grows the white area in the modified mask P by marking the 
pixel p' 5 as a "white - pixel atthe step 108. . - u w fc 

[00581 At the step 109, the color replacing module 18 determines if all pixels within the mask Phave been checked. 
If not, then the step 104 is repeated. If so, the step no is the next step at which the masks Pand Pare merged by a 

to binary OR operation into a newer version of P. The process then returns to the step 102. 

roosgi if at the step 102, it is determined that r> is greater than N. the step 111 is performed at when the color 
replacing module 18 receives one pixel p* from the newer version of P. Then the step 112 is performed to determine 
if ihe pixel is a -white- pixel (i.e., 1). if not, the step 114 is performed- if the answer is positive, then the color 
replacing module 1 8 changes the color of the pbcel to monochrome (grey) or other predefined colors at the step 11 & 

is m one embodiment, this is done by changing the luminance and chrominance values of the pixel The luminance value 
can be changed for example, by a 0.8 factor and the chrominance values can be set TO zero. Then the step n 4 is 
performed at which the color replacing module 18 detennines if all of the pixels within the newer version of P have 
been checked. If so. the process ends at the step 115. If no, the step 111 is repeated. t 
[0060] The disclosures in United States patent application no. 06/956,380, irom which this application claims pnority, 

so and in the abstract accompanying this application are incorporated herein by reference. 



Claims 

25 1. A red-eye reduction system, comprising; 

(A) a masking module (16) operable to convert an image (11) into a mask having first state areas representing 
red colour pixels and second state areas representing other colour pixels, wherein the image (11) includes an 
eye with a red pupil; ^ . 

so (B) a pupil locating module (17) coupled to the masking module (16) operable to locate a substantially first 

state area in the mask that resembles a pupil; 

(C) a colour replacing module (1 6) coupled to the pupQ locating module (1 7) operable to change the red colour 
pixels in the area into a predetermined colours wherein the colour replacing modulo (18) is also operable to 
adjust boundary of the area by changin g th e colours of pixels in close piraimity to the area to the predetermined 
ss colour if the colour of these pixels is determined to be sufficiently close to red. 

£ A red-eye reduction system as in claim 1 , wherein the pupil locating module <17) is operable to detect if the sub- 
stantially first state area resembles a pupil by determining if the substantially first state area Is a substantially 
largest first state area within the mask, wherein after determining that the substantially first state area resembles 
40 a pupil, the pupil locating module (17) is operable to modify the mask by changing all otherfJrst state areas in the 
mask into second state such that ihe modified mask only has the substantially first state area. 

3. A red-eye reduction system as in claim 1 . wherein the substantially first state area includes a second state area 
within the first state area, wherein the second state area represents the glint in the pupil, wherein the predetermined 
colour is a zero chrominance, wherein the colour replacing module (18) is operable to change the luminance value 
of all pixels within the substantially first state area by a predetermined factor. 

4. A red-eye reduction system as in any one of claims 1 to 3. wherein the masking module (16) is operable to convert 
the image into the mask by first raising red-green-blue values of the image to a predetermined power and then to 

so perform a standard luminance-chrominance conversion. 

5. A red-eye reduction system as in any one of claims 1 to 4, wherein the colour replacing module (IB) is operable 
to determine if the colour of the adjacent pbcel is sufficiently close to red by comparing the colour value of the 
adjacent pixel with (i) a median red value and (ii) a median green value of all the red colour pixels within the 

ss substantially first state area. 

a A red-eye reduction system as in any one of claims 1 to 5, wherein the adjacent pixel is a pfceJ that has at least 
one neighbouring pixel in the substantially first state area. 
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7. A method of reducing red-eye effect in a digital image comprising: 

(A) converting an image (11 ) having an eye with a red pupil into a mask having first state areas representing 
red colour pixels of the image and second state areas representing other colour pixels of the image: 
s (B) processing the mask to locate a substantially first state area in the mask that resembles a pupil: 

(C) changing the red colour pixels in the area into a predetermined colour with boundary adjustments such 
that the colour of pixels in close proximity to the area is also changed to the predetermined colour if the colour 
of these pixels is determined to bo sufficiently close to red. 

to 8. A method as In claim 7. wherein the step (B) comprises the steps of: 

0) determining if the substantially first state area is a largest substantially first state area in the mask; 
(li) determining if the substantially first state area is substantially circular; 

(III) modifying the mask by changing all other first state areas in the mask into second state such that the 
is modified mask only has the substantially first state area. 

9. A method as in claim 7 or 8, wherein the step (C) comprises the steps of: 

(a) changing the colour of the red colour pixels within the substantially first state area to the predetermined 
20 colour by setting the chrominance of the red colour pixels to zero; 

(b) changing the luminance value of all pixels within the substantially first state area by a predetermined factor. 

10. A method as in claim 9, wherein the step (A) comprises Jthe steps of: 

25 ( a ) raising red-green-Wue vaJues of the image to a predetermined power: 

(b) performing a standard iurninance-chrominance conversion; 

(c) creating the mask. 
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(57) A red-eye reduction system (10) includes a 
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a mask having first state areas representing red colour 
pixels of the image and second state areas representing 
other colourpixels of the image. "The image (11) includes 
an eye with a red pupil. A pupil locating module (17) is 
coupled to the masking module (1 6) to locate a substan- 
tially first state area in the mask that resembles a pu pil. 



A colour replacing module (1B) is then coupled to the 
pupil locating module (1 7) to change the red colour pb> 
els in the area into monochrome (grey) or other prede- 
fined colours. The colour replacing module (1 8) also ad- 
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of pixels in close proximity to the area if the colour of 
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